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Abstract 

Nursing  homes  have  lagged  in  the  development  and  use  of  technology  and  clinical 
informatics.  This  paper  describes  a  practical  model  of  translating  clinical 
informatics  research  into  practice.  The  Minimum  Data  Set  (MDS)  assessment 
data  collected  by  nursing  homes  nationwide  is  translated  into  knowledge-based 
information  that  supports  continuous  quality  improvement.  It  does  so  by 
providing  timely  Web-based  reports  alerting  staff  to  the  likelihood  of  an  adverse 
outcome,  along  with  individualized  resident  risk  profdes  to  guide  preventive  care 
plan  development.  The  adverse  outcomes  addressed  in  this  study — falls  and 
pressure  ulcers — are  associated  with  considerable  morbidity  and  mortality  and 
represent  serious  quality  of  care  issues  for  the  elderly  nursing  home  population. 
These  events  are  usually  preventable  yet  contribute  significantly  to  the  growing 
costs  of  health  care,  insurance,  and  liability.  This  paper  describes  the  risk  reports 
and  how  nursing  home  staffs  are  using  them,  barriers  to  use  of  clinical 
informatics,  measurable  changes  in  processes,  outcomes  and  quality  of  care,  and 
implications  for  other  Web-based  decision-support  systems  in  long  term  care 
settings. 


Introduction 

This  study,  which  started  in  October  2001,  is  one  of  the  22  patient  safety 
projects  funded  in  the  first  round  by  the  Agency  for  Healthcare  Research  and 
Quality  (AHRQ)  in  the  area  of  Clinical  Informatics  to  Promote  Patient  Safety 
(CLIPS),  and  is  one  of  seven  in  the  area  of  long  term  care  (LTC).  The  goal  of  the 
project  is  to  determine  whether  adverse  outcomes  for  nursing  home  residents  can 
be  prevented  through  the  use  of  prospective  Web-based  risk  management  reports. 

This  project  supports  the  use  of  clinical  informatics  by  “front  line”  nursing 
staff.  Through  proactive  use  of  available  electronic  clinical  data,  we  shift  the 
focus  from  using  the  extensive  Minimum  Data  Set  (MDS)  ’  ~  assessment  data 
designed  to  investigate  outcomes  after  they  occur  to  one  focusing  on  preventive 
actions. 

Many  adverse  outcomes  are  preventable  and  occur  due  to  health  care  staffs 
limitations  as  “data  processors.”3’4' 5  Preventing  poor  outcomes  requires 
committing  more  time  to  processing  patient  data,  but  nursing  staff  are  too  busy  to 
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consistently  analyze  and  detect  the  multitude  of  conditions  specified  by  the 
growing  body  of  protocols  and  standards  of  care.  Many  studies  have  shown  that 
these  protocols  are  neither  well-known  nor  consistently  followed  by  nursing  home 
staff6’ 7' 8  Nursing  staff  are  plagued  with  too  much  data  and  too  little  information. 
They  need  access  to  the  right  information  at  the  right  time  to  use  it  effectively  to 
improve  outcomes.  Fonner  Secretary  of  the  Department  of  Health  and  Human 
Resources  Tommy  Thompson  stated  the  problem  succinctly  in  2003:  “One  of  our 
challenges  is  the  explosion  of  new  knowledge  resulting  from  research,  which  has 
surpassed  the  ability  of  individual  practitioners  to  absorb  and  apply  it  while 
actually  delivering  care.  This  knowledge  is  only  as  useful  as  the  ability  of  a 
provider  to  remember  it  when  it  really  matters.”9 

In  today’s  long  term  care  environment  of  shrinking  finances,  staffing 
shortages,  high  turnover,  increasing  workloads,  and  growing  acuity  levels,10' 11 
nursing  staff  cannot  gather  and  analyze  the  resident  data  needed  to  accurately 
assess  risk  without  the  aid  of  computerized  decision-support  tools.  This  project 
translates  evidence-based  research  into  practice  through  proactive  risk 
management  reports  that  1)  synthesize  knowledge  derived  from  current  research, 
up-to-date  guidelines  and  standards  of  care,  successful  protocols,  and  best 
practices;  2)  apply  this  knowledge  to  existing  MDS  data;  and  3)  display  risk 
information  in  useful  and  readily  accessible  formats  that  can  be  acted  on  to  avoid 
adverse  outcomes  and  improve  systems  of  care. 

Since  most  nursing  homes  have  not  had  access  to  clinical  informatics 
applications  designed  to  automate  manual  processes,  ~  scant  information  exists 
regarding  the  use  of  informatics  and  decision-support  systems  to  improve  process 
and  outcomes  of  long  term  care.13, 14 

The  adverse  outcomes  addressed  in  this  paper — falls  and  pressure  ulcers — 
represent  serious  quality  of  care  issues  in  nursing  homes  nationwide  and 
contribute  significantly  to  higher  liability  and  health  care  costs.  Bishop  et  al. 
observed,  “Medicare  is  spending  billions  to  treat  preventable  (fall)  injuries... at  an 
average  cost  of  $  1 ,272  per  incident. .  .yet  interventions  are  not  widely 
disseminated.  Medicare  could  realize  substantial  savings  if  these  injuries  could  be 
prevented.”15  Prevention  of  pressure  ulcers  has  tremendous  financial 
implications — the  cost  of  treating  a  pressure  ulcer  is  estimated  at  2.5  times  the 
cost  to  prevent  one.16  Estimated  yearly  expenditures  on  pressure  ulcers  amount  to 
$7.5  billion.17 

Falls  and  pressure  ulcers  are  often  avoidable.  Fall  intervention  programs 

focused  on  individuals  most  at  risk  can  reduce  fall  rates  substantially.  Ginter  and 

Minn  suggest  that  fall  prevention  programs  must  focus  on  resident-specific  risk 

factors  and  target  interventions  to  the  individual.18  Meanwhile  Rubenstein19  and 
*20 

Capezutr  emphasize  that  the  most  cost-effective  strategy  for  preventing  falls  is 
to  identify  high-risk  individuals.  The  American  Medical  Directors  Association 
issued  similar  advice,  “Medical  directors  and  administrators  would  be  wise  to 
develop  and  implement  a  comprehensive,  facility-wide  process  for  determining 
causes  and  assessing  risks  of  falls.  Otherwise,  caregivers  may  miss  important 
diagnostic  clues,  thus  bypassing  opportunities  to  correct  modifiable  risk  factors 
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and  avert  subsequent  falls.”  Other  studies  have  similarly  concluded  that 
modifiable  risk  factors  predispose  residents  to  development  of  pressure  ulcers15 
but  recent  research  provides  evidence  that  valid  and  reliable  pressure  ulcer  risk 
assessment  tools  are  seriously  underused  and  prevention  and  that  treatment 
guidelines  are  rarely  implemented.4 

The  risk  assessment  and  care  planning  information  provided  to  nursing  home 
staff  through  this  study  is  based  on  the  standardized  MDS  assessment  tool.  The 
MDS  is  completed  on  admission  and  at  least  quarterly  thereafter  for  all  residents. 
Furthermore,  it  has  been  made  electronically  accessible  since  1999,  making 
comprehensive  clinical  information  available  for  technology  applications. 

In  spite  of  the  potential  of  this  rich  data  set,  the  MDS  has  not  been  widely 
used  for  risk  assessment,  in  part  because  the  data  have  been  largely  inaccessible  to 
nursing  staff  in  readily  available  and  useful  formats.  Integral  to  the  goals  of  this 
project  was  the  potential  to  reduce  the  time-consuming  redundancy  involved  in 
manually  collecting  patient-level  data  by  demonstrating  to  nursing  staff  that  the 
MDS  can  be  used  as  the  primary  data  source  for  risk  assessment.  Jogerst,  et  al. 
concluded  that  “the  MDS  is  underutilized — better  tools  to  provide  MDS 
information  could  enhance  physician  and  clinical  practice  in  nursing  homes  by 
relaying  valuable  information  to  provide  better  care  to  the  patient.”  “  Studies  have 
shown  MDS  data  can  be  used  to  accurately  assess  risk  for  adverse  outcomes.  Vap 
and  Dunaye  compared  the  MDS  with  the  widely  used  Braden  Scale  and  found 
that  eight  MDS  items  predicted  pressure  ulcers  more  accurately.14 

Existing  risk  assessment  tools  are  inadequate  due  in  part  to  the  fact  that  they 
were  developed  years  ago  and  do  not  utilize  current  technology  and  database 
software  capable  of  quickly  synthesizing  and  analyzing  large  amounts  of  data  to 
produce  information  in  formats  that  are  useful  to  busy  nursing  staff.  These  tools 
have  many  limitations,  including  the  following:  (1)  they  require  staff  to  manually 
collect  information  from  various  sources;  (2)  they  employ  a  limited  set  of  risk 
factors  that  can  be  easily  captured  and  immediately  assimilated  on  paper,  and  thus 
do  not  consider  many  co-morbid  conditions  known  to  be  highly  predictive  of 
adverse  events;  (3)  they  usually  weight  each  risk  factor  equally  or  very  crudely, 
when  in  fact  certain  responses  are  far  more  predictive  of  risk;  (4)  they  do  not 
capture  resident  history,  e.g.,  the  cumulative  effects  of  chronic  conditions  and 
diagnoses  that  contribute  to  risk;  (5)  they  fail  to  weigh  interactions  of  smaller  risk 
factors  that  add  up  to  high  risk;  and  (6)  they  often  are  not  validated. 


Method 

Study  design 

All  of  New  York  State’s  (NYS)  650+  nursing  homes  were  invited  to 
participate  in  this  research  project.  Approximately  150  (23  percent)  nonprofit, 
proprietary,  and  public  facilities  volunteered.  Eighty  were  initially  selected,  and 
1 1  more  nursing  homes  were  added  within  the  first  8  months,  for  a  total  of  91 
participating  facilities. 
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Since  the  only  technology  requirement  for  participation  was  Internet  access, 
which  is  required  for  MDS  data  submission,  very  few  facilities  were  unable  to 
participate.  Volunteering  seemed  to  be  more  closely  related  to  resources  vs. 
workloads  (including  other  quality  improvement  projects)  as  well  as 
management/staff  interest  and  willingness  to  participate  in  outcomes  research 
requiring  ongoing  commitment  of  staff  time  and  changes  in  processes. 

The  study  group  of  9 1  nursing  homes  selected  from  the  volunteer  group  was 
fairly  representative  of  NYS  nursing  homes,  but  these  homes  also  had  higher  than 
average  pre-intervention  quality  measures  for  falls  and  pressure  ulcers  (more  falls 
and  pressure  ulcers  than  expected  after  risk-adjustment).  They  were  purposely 
selected  to  ensure  that  participating  facilities  had  opportunities  to  improve  (Table 
1).  The  91  participating  nursing  homes  were  slightly  larger  (mean  bed  size  of  242 
vs.  170  for  non-volunteers)  and  were  more  likely  to  be  nonprofit  or  public  (81 
percent  vs.  45  percent  that  did  not  volunteer)  because  a  much  larger  proportion  of 
volunteer  facilities  consisted  of  members  of  the  New  York  Association  of  Homes 
and  Services  for  the  Aging,  where  the  research  was  conducted.  Importantly,  the 
mean  fall  and  pressure  ulcer  risk  scores  were  similar  for  all  three  groups. 

As  of  spring  2004,  66  facilities  (>70  percent)  are  regularly  accessing  the  risk 
reports  and  considered  active  participants  in  the  study.  The  high  retention  rate  is 
indicative  of  the  perceived  quality  and  usefulness  of  the  risk  reports. 


Table  1.  Comparison  of  New  York  State  nursing  homes  that  volunteered  for  the  project 
(by  selection  status)  with  those  that  did  not  volunteer 


Volunteered  Did  not  volunteer 

Selected 

Not  selected 

Number 

00 

CD 

64 

512 

Mean  bed  size 

242 

176 

170 

%  Nonprofit/voluntary 

81 

65 

45 

%  Downstate  (NYC/LI) 

31 

17 

42 

Mean  fall  risk  score 

54 

57 

50 

Adjusted  Fall  Quality  Measure** 

2.3 

-.47 

-1.6 

Mean  PU  risk  score 

45 

47 

45 

Adj.  PU  quality  measure** 

.55 

-1.1 

-.2 

Mean  length  of  stay,  quarters 

9.8 

10.0 

9.8 

NYC=New  York  City;  LI=Long  Island;  PU=Pressure/Ulcer 
‘Data  not  complete  for  2  of  the  91  selected. 

“This  score  represents  difference  between  actual  and  expected  number  of  falls  and  pressure 
ulcers  in  resident  population  (excluding  short-stay  residents)  after  risk-adjustment,  averaged 
across  the  facilities  for  4  quarters  before  the  project  began.  A  positive  score  indicates  the  facility 
has  more  adverse  events  than  expected  and  thus  a  poorer  quality  of  care  measure.  A  negative 
score  indicates  a  better  quality  of  care  measure. 
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Results 

Risk  assessment  models  using  longitudinal  data 

The  risk  reports  are  based  on  predictive  regression  models  developed  using 
longitudinal  MDS  data  for  all  residents  in  NYS  nursing  homes  from  January  2001 
through  December  2002.  The  data  for  these  approximately  138,000  residents  was 
split  randomly  into  development  and  confirmation  files.  Risk  factor  weights,  both 
positive  and  negative,  were  summed  to  calculate  indices  for  diseases,  nutrition, 
chronic  conditions,  infections,  mobility,  and  other  MDS-based  measures  that, 
along  with  historical  indices  of  prior  adverse  events,  constitute  the  set  of 
polychotomous  independent  variables.  Logistic  regression  was  used  to  model  the 
risk  of  an  adverse  event  in  the  next  quarter.  Model  accuracy  was  evaluated  using 
the  confirmation  file. 

The  models  performed  very  well — the  risk  reports  accurately  predicted  8 1 
percent  of  falls  and  70  percent  of  pressure  ulcers  (i.e.,  of  residents  who  had  a  fall 
recorded  on  their  next  assessment,  8 1  percent  were  identified  as  “high  risk”  using 
their  current  assessment)  (Table  2,  sensitivity).  The  measures  of  concordance  of 
the  model,  viz.,  C  statistics  are  .88  and  .85  respectively. 

Tools  with  lower  predictive  accuracy  identify  more  residents  to  be  at  high 
risk,  while  our  more  precise  tools  show  them  to  be  at  lower  risk.  Our  models 
target  far  fewer  residents.  While  a  typical  pressure  ulcer  risk  tool  places  30-50 
percent  of  residents  at  “high  risk,”"  our  methodology  places  only  25  percent  of 
residents  at  “very  high”  or  “high”  risk.  Much  is  lost  when  risk  is  presented  in  a 
“present”  or  “absent”  concept  (a  style  many  manual  tools  feature)  because 


Table  2.  Logistic  regression  model  statistics  for  falls  and  pressure  ulcers 


Model 

Performance  Statistics 

Data  set 

Development 

Confirming 

Falls 

C  statistic 

.883 

na 

Detected  ( sensitivity ) 

81.1% 

81.2% 

Positive  predictive  value 

75.4% 

75.9% 

False  negative 

6.7% 

6.6% 

False  positive 

9.3% 

9.0% 

Correct  prediction  (y/n) 

84.1% 

84.5% 

Pressure  Ulcers 

C  statistic 

.853 

na 

Detected  ( sensitivity ) 

69.7% 

69.2% 

Positive  predictive  value 

67.6% 

66.3% 

False  negative 

8.0% 

7.9% 

False  positive 

8.8% 

9.0% 

Correct  prediction  (y/n) 

83.4% 

83.1% 
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pressure  ulcer  risk  is  located  along  a  continuum.  Failure  to  accurately  differentiate 
levels  of  risk  creates  significant  unnecessary  costs  for  facilities. 

The  risk  models  in  the  present  study  also  keep  false  positive  and  false 
negative  rates  very  low  (less  than  10  percent,  Table  2).  Tools  with  low  sensitivity 
miss  many  residents  who  are  actually  at  high  risk,  leading  to  development  of 
avoidable  pressure  ulcers,  higher  costs  of  care,  increased  exposure  to  state  and 
federal  sanctions,  litigation,  and  other  severe  problems  for  the  facility.24 

Our  models  are  able  to  more  accurately  calculate  the  likelihood  of  a  future  fall 
or  pressure  ulcer  and  to  better  identify  residents  at  the  greatest  risk  than  those 
developed  in  earlier  studies  due  to  several  factors: 

1 .  Use  of  a  database  methodology  that  links  resident  assessments  for  up 
to  eight  quarters  to  create  a  rich  longitudinal  perspective  and  to  capture 
important  risk  factors  not  coded  on  the  resident’s  most  recent 
assessment,  such  as  a  history  of  the  outcome  and  chronic  diseases. 
Tinetti  noted,  “The  majority  (of  falls)  result  from  interactions  between 
long-term  and  short-term  predisposing  factors.”"  While  previous 
findings  in  this  area  have  been  based  primarily  on  cross-sectional 
studies,  researchers  have  found  that  a  longitudinal  approach  better 
identifies  addressable  causes  for  other  adverse  outcomes.26 

2.  Our  models  do  not  rely  on  a  limited  set  of  risk  factors  and  simple 
“check-offs”  used  by  existing  manual  tools,  which  check  only  whether 
select  conditions  are  present  or  not. 

3.  Our  models  expand  the  power  of  the  MDS  through  the  use  of  existing 
and  newly  developed  indices  comprised  of  multiple  MDS  items 
weighted  and  summed  to  create  broad  risk  variables  (e.g.,  diseases, 
cognitive  status,  mobility,  and  medications).  Support  for  measures 
based  on  multi-item  scales  was  provided  by  Mor  et  al.“  Typically, 
logistic  regression  models  employ  binary  independent  variables  with 
high  odds  ratios,  but  this  approach  limits  the  number  of  factors  that 
can  be  applied  and  excludes  variables  that  lower  the  likelihood  of  an 
adverse  event.  For  example,  a  validated  scale  of  cognitive  status,  the 
Cognitive  Perfonnance  Score  (CPS),  has  been  constructed  using  MDS 
data.'  ’  Using  the  CPS,  we  find  that  “moderately  impaired”  residents 
are  30  percent  more  likely  to  fall  than  are  more  severely  impaired 
residents.  However,  nursing  home  staff  cannot  calculate  the  CPS 
manually.  Furthermore,  the  relationship  between  level  of  cognitive 
impairment  and  falls  is  not  obvious  to  them.  Utilizing  MDS  scales 
such  as  the  CPS  in  the  risk  models  greatly  enhances  their  accuracy  and 
use  for  risk  assessment. 

Intervention 

Three  primary  types  of  reports  have  been  developed  and  made  available  for 
immediate  display  and  printing  at  the  nursing  home  site  via  a  secure  Internet 
connection: 
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1 .  At  risk  reports.  These  identify  residents  by  level  of  risk  for  a  fall  or 
pressure  ulcer,  organized  by  unit  (Figure  1). 

2.  Resident  risk  profile  reports.  Individualized  resident  level  list  of 
specific  risk  factors  (by  level  of  importance)  allowing  nursing  staff  to 
make  individualized  care  decisions  based  on  data-driven,  knowledge- 
based,  and  resident-focused  information  (Figure  2). 

3.  Feedback  reports.  In  the  next  period,  compare  (1)  actual  results  to 
previous  quarter  predictions  of  risk  of  the  adverse  outcome  (Figure  3); 
(2)  avoided  (prevented)  adverse  outcomes  (resident  identified  as  very 
high  risk  and  adverse  event  did  not  occur);  and  (3)  “unexpected”  or 
potentially  avoidable  adverse  outcomes  (i.e.,  resident  was  “low  risk” 
based  on  comprehensive  risk  model,  but  adverse  event  occurred). 

The  reporting  software  was  carefully  designed  to  be  user-friendly — the 
screens  are  logical  and  easy  to  follow.  The  ability  to  provide  information  by  unit 
within  the  facility  greatly  increases  the  immediate  usefulness  of  the  reports.  Units 
may  have  a  unique  resident  population  (such  as  Hospice  or  Alzheimer’s)  or 
operate  differently,  thus  the  software  helps  to  target  “real  problems”  on  specific 
units. 

The  feedback  reports  were  added  to  foster  greater  trust  and  use  of  the  reports. 
Facility  staff  can  see  that  the  models  assign  risk  as  accurately  as  or  better  than 
their  manual  tool,  and  that  a  significant  proportion  of  the  residents  at  high  risk 
actually  do  experience  the  event.  Staff  can  view  the  risk  profiles  for  residents  who 
had  the  outcome  to  see  if  they  addressed  all  the  risk  factors  in  care  plans  and 
interventions.  They  can  also  better  understand  how  they  were  able  to  prevent 
adverse  outcomes  in  high-risk  residents  and  use  this  information  to  modify 
interventions  facility-wide.  Over  time,  it  is  expected  that  the  number  of  adverse 
events,  particularly  falls,  will  decrease  if  these  reports  are  effectively  used  to 
accurately  identify  high  risk  residents,  plan  patient-centered  interventions,  and 
monitor  the  results  of  the  interventions. 

Comparison  of  informatics-based 
risk  reports  with  manual  risk  tools 

The  feedback  provided  by  many  participants  showed  that  the  risk  levels 
assigned  by  the  models  closely  conformed  to  and  frequently  outperformed  their 
manual  tool.  In  fact,  many  have  replaced  their  manual  tool  completely  with  the 
risk  reports.  We  conducted  a  study  of  55  residents  in  four  nursing  homes  to 
compare  assigned  levels  of  risk  for  pressure  ulcers  using  the  risk  model  vs.  the 
manually  scored  Braden  Scale.  The  risk  model  resulted  in  a  much  smaller 
proportion  of  residents  identified  as  high  risk  (16  percent  using  risk  reports  vs.  42 
percent  flagged  by  the  Braden  Scale).  However,  it  achieved  much  higher 
prediction  accuracy  (38  percent  of  high  risk  residents  on  risk  report  experienced  a 
pressure  ulcer  in  the  next  quarter  vs.  only  13  percent  of  those  flagged  by  the 
manual  tool)  and  a  much  lower  false  positive  rate  (11  percent  using  risk  reports 
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Pressure  ulcer  risk  report  -  All  residents  except  new  admissions 
Based  on  assessments  for  reporting  period  12/01/03  to  04/20/04 

(Click  underlined  column  heading  [e.g.  Unit]  to  sort  by  that  heading;  click  on  resident  name  to  view  risk  profile.) 
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i8a  =  Primary  reason  for  assess  ment  for  long  stay  resident  based  on  federal  requirements. 

v8b  =  Type  of  assessment  requird  for  Medicare  Prospective  Payment  System  (PPS)  for  short-sta  and/or  post-acute  residents. 


Figure  3.  Feedback  Report  1  -  Actual  pressure  ulcers  vs.  predicted  risk  in  previous  quarter 
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vs.  46  percent  using  the  Braden  tool).  Critically,  no  resident  whom  the  manual 
tool  put  at  high  risk  and  whom  the  risk  model  put  at  lower  risk  had  a  pressure 
ulcer  in  the  subsequent  quarter.  Clearly,  a  tool  that  places  such  a  high  proportion 
of  residents  at  high  risk  is  problematic  and  constitutes  an  inefficient  use  of  limited 
staffing  resources. 

Utilization  of  risk  reports 

In  the  initial  phase  of  the  study,  participating  nursing  home  staff  attended  a 
one-and-a-half-day  training  session  to  fill  key  knowledge  gaps  in  the  use  of 
clinical  informatics  and  to  understand  and  use  the  risk  reports.  Topics  included 
basic  statistical  concepts  (benchmarking  and  trend  analysis),  how  to  interpret 
outcomes  data,  and  how  to  use  the  reports  to  plan  targeted  remediation  and 
change  system-wide  care  processes.  Throughout  the  project,  participants  have 
received  ongoing  communication  and  support  through  annual  workshops  and 
phone/e-mail  communications  from  project  staff  and  nurse  consultants. 

Short  surveys  were  conducted  during  2003,  with  followup  contacts  between 
September  2003  and  March  2004,  to  document  how  participants  were  using  the 
reports.  The  contacts  were  open-ended,  allowing  facility  staff  to  openly  describe 
both  how  they  implemented  the  reports  and  problems  encountered. 

Utilization  of  the  reports  is  monitored  through  an  application  that  captures 
visits  to  the  report  web  pages.  This  allows  project  staff  to  identify  which  facilities 
(and  individuals)  regularly  access  the  reports  and  which  reports  are  used  most. 

Based  on  utilization  and  survey  feedback,  the  91  participating  nursing  homes 
were  stratified  into  four  groups: 

1.  High  access  and  high  integration  with  care  planning  (N=18; 

20  percent  of  total  participants).  These  facilities  regularly  run  and  use 
all  risk  reports  (averaging  10  times  per  month),  and  use  them 
prospectively  (proactive  care  planning)  rather  than  solely 
retrospectively  (e.g.,  to  investigate  causes  of  a  fall).  They  rate  the  risk 
reports  as  very  useful. 

2.  Moderate  to  high  access  and  some  integration  with  care  planning 

(N=15;  16  percent).  These  facilities  access  the  risk  reports  less  often 
(5-10  times  per  month).  They  primarily  use  risk  level  reports,  and  use 
individualized  risk  profiles  less  often.  They  took  longer  to  obtain  staff 
“buy-in”  and  often  had  to  prove  that  the  new  infonnatics  tool  worked 
as  well  as  or  better  than  existing  risk  assessment  tools. 

(NOTE:  Groups  1  and  2  [33  homes;  36%  of  participants]  will  be  used 
to  define  “participating  users”  for  the  project  evaluation  study 
discussed  below.) 

3.  Low  to  high  access  but  little  integration  with  care  planning  (N=33; 
36  percent).  These  facilities  run  the  reports,  but  do  not  fully 
understand  the  information  and  how  it  might  be  used.  The  risk  profiles 
are  used  to  ensure  that  all  risk  factors  are  addressed  in  the  care  plan; 
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reports  do  not  impact  care  in  a  proactive  way  and  are  viewed  as 
adjunct  or  extra  paperwork. 

4.  Little  or  no  access  of  the  reports  (N=25;  27  percent).  These  facilities 
never  attempted  to  use  risk  reports,  most  often  due  to  staff  changes. 

Based  on  our  analyses  of  utilization  and  survey  feedback,  and  considering  the 
success  of  programs  using  advanced  practice  nurses  to  work  with  nursing  home 
staff  to  achieve  quality  improvements,30’ 31  we  determined  it  was  necessary  to 
deploy  more  nurse  consultant  support  to  assist  participants  in  using  and 
interpreting  the  reports.  In  fall  2003,  two  experienced  registered  nurse  consultants 
began  regularly  contacting  staff  to  help  them  use  the  risk  reports.  The  nurses  have 
concentrated  on: 

1 .  Gathering  detailed  information  on  how  successful  facilities  are  using 
the  risk  reports  and  how  the  reports  are  impacting  care  processes  and 
outcomes.  The  best  practices  were  incorporated  into  a  “Step-By-Step 
Guide”  that  was  shared  with  staff  who  were  not  familiar  with  using  the 
reports. 

2.  Providing  intensive  support  (primarily  via  telephone,  some  on-site)  to 
facilities  whose  report  use  was  low  but  whose  staff  expressed  a  desire 
to  increase  use.  The  support  included  practice  exercises  designed  to 
overcome  barriers  to  using  technology  and  demonstrations  of  ways  the 
reports  can  become  a  regular  part  of  the  care -planning  process. 

Earlier  studies  found  that  on-site  support  was  important  when  implementing 
and  sustaining  a  new  quality  improvement  intervention.32, 33  The  nurse  support 
provided  in  this  project  relied  primarily  on  telephone  support  and  electronic 
communication  rather  than  on-site  support.  Importantly,  we  found  that  while  the 
level  of  required  support  has  decreased  over  time,  the  level  of  use  of  the  reports 
has  remained  relatively  stable.  This  is  a  positive  finding  related  to  sustainability 
of  the  intervention;  the  nurse  consultants  focused  on  institutionalizing  the  use  of 
risk  reports  by  providing  guides,  tools,  and  best  practices  in  integrating  the  reports 
into  practice. 

Barriers  to  use  of  clinical  informatics 

An  important  variable  impacting  facility  use  and  integration  of  clinical 
informatics  tools  such  as  the  computerized  risk  reports  is  consistent  staff  who 
understand  and  use  the  information.  Staff  turnover  can  significantly  impact  the 
implementation  of  quality  improvement  programs  and  interventions,  including  the 
use  of  new  technology,  and  this  project  is  no  exception.  While  actual  staff 
turnover  data  have  not  yet  been  gathered,  the  project  nurse  consultants  regularly 
educated  new  facility  staff  about  the  risk  reports. 

In  addition  to  the  well-documented  workforce  shortages,  high  turnover,  and 
high-stress  environments  of  nursing  home  staff,  one  of  the  most  common  barriers 
reported  was  that  front  line  staff  were  not  as  responsive  to  the  reports  as  staff 
involved  in  facility-wide  quality  improvement.  Unit  charge  nurses  were  more 
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likely  to  see  reports  as  additional  work  and  were  reluctant  to  review  or  share  them 
with  direct  care  staff. 

Many  in  the  nursing  staff  displayed  a  traditional  reluctance  to  change,  and 
some  simply  dismissed  the  reports  as  duplicative  of  tools  they  already  had  in 
place.  Though  many  acknowledged  that  the  new  risk  reports  were  probably  more 
accurate,  changing  tools  and  protocols  required  investing  time  and  resources  and 
did  not  seem  worth  the  effort  to  many  nurses. 

Successful  integration  of  clinical  infonnatics  into  organizational  process 
occurs  when  (1)  the  evidence  matches  professional  consensus  and  patient  needs; 
(2)  the  organization  is  receptive  to  change  with  strong  leadership  and  appropriate 
monitoring  and  feedback  systems  in  place;  and  (3)  there  is  appropriate  facilitation 
of  the  change.34  All  of  these  conditions  were  met  in  the  nursing  homes  that  had 
high  integration  and  use  of  the  reports.  Almost  all  had  a  leader  or  change  agent 
(often  the  administrator  or  director  of  nursing  or  quality  improvement)  who 
believed  that  the  reports  would  be  useful  and  cost-effective,  encouraged  (or 
required)  that  they  be  used,  and  followed  up  on  their  use. 

As  one  example  of  this  process,  the  administrator  of  a  facility  with  low  use  of 
the  risk  reports  for  almost  a  year  conferred  with  a  nurse  consultant  (facilitator) 
and  agreed  to  use  the  reports  as  their  sole  risk  assessment  tool  on  a  trial  basis.  He 
encouraged  the  staff  to  use  the  reports,  and  after  only  3  months  they  determined 
the  reports  provided  more  accurate  and  complete  risk  assessment  and  care 
planning  information.  They  were  willing  to  fully  integrate  them  into  practice  and 
replace  their  manual  tools. 

Finally,  there  is  an  underlying  “fear  of  the  unknown”  related  to  (1)  how  the 
surveyors  will  react  to  use  of  this  infonnation,  and  (2)  the  potential  of  added 
liability  of  having  access  to  this  type  of  information.  (For  example,  staff 
frequently  expressed  concerns  about  having  access  to  risk  reports  giving  them 
prior  knowledge  that  a  resident  is  at  “very  high  risk”  for  development  of  a 
pressure  ulcer  and  a  comprehensive  list  of  addressable  risk  factors.  What  if  the 
information  is  not  fully  acted  on  and  a  pressure  ulcer  develops?)  These  fears  are 
very  real  to  long  term  care  nurses  with  the  recent  dramatic  growth  in  litigation. 

Successful  use  of  risk  reports  in  practice 

Nursing  staff  in  facilities  with  high  and  low  use  of  risk  reports  have  been 
interviewed  to  better  understand  the  dynamics  that  impact  use  of  the  risk  reports 
and  barriers  encountered.  All  of  the  33  homes  regularly  using  the  reports  have 
provided  feedback.  The  conditions  for  successful  use  in  practice  include:  (1) 
administrative  level  and  nursing  staff  buy-in  and  support,  (2)  development  of  an 
actual  process  integrating  the  risk  reports  into  ongoing  quality  improvement 
processes,  and  (3)  a  facility  “champion”  to  keep  the  effort  focused  and  on  track. 
Several  nursing  homes  were  unhappy  with  or  evaluating  their  current  risk  tools 
when  the  project  started,  and  thus  were  more  likely  to  accept  and  use  the  reports 
as  their  primary  risk  tool. 
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Other  factors  found  to  facilitate  use  of  decision-support  tools  in  health  care, 

•7  ^ 

such  as  the  risk  reports,  have  been  documented  in  other  studies. 

1 .  Nursing  home  staff  perceived  the  data  as  useful,  meaningful,  and  of 
high  enough  quality  to  motivate  them  to  change. 

2.  Staff  took  the  time  to  validate  and  promote  the  credibility  of  the  data. 

3.  Use  of  benchmarks  and  regular  monitoring  improved  the 
meaningfulness  of  the  information. 

4.  Leaders  enhanced  the  effectiveness  of  the  information. 

5 .  Support  of  the  system  was  sustained  long  enough  to  improve 
performance. 

In  the  33  “user”  facilities  (defined  as  Groups  1  and  2  above),  the  risk  reports 
are  used  in  a  wide  variety  of  ways.  Most  commonly  they: 

•  Serve  as  the  primary  risk  assessment  tool  (many  replaced  their  manual 
tools). 

•  Guide  development  of  care  plans  and  interventions  aimed  at 
prevention. 

•  Ensure  comprehensive  assessment  of  all  potential  resident  risk  factors. 

•  Educate  and  inform  interdisciplinary  team,  nurse  managers,  and 
certified  nurse  assistants. 

•  Support  committee  activities  (e.g.,  quality  improvement,  fall,  and  skin 
care). 

•  Support  responses  to  survey  team  (e.g.,  help  demonstrate  that  adverse 
event  was  unavoidable). 

Potential  for  success  in  practice — a  case  study 

A  300-bed  facility  with  consistently  high  use  of  the  fall-risk  reports  provides  a 
case-study  example  of  how  risk  reports  can  be  used  successfully.  The  facility  (1) 
replaced  their  manual  risk  assessment  tool  and  fully  integrated  the  risk  reports  in 
their  quality  improvement  program  for  falls,  and  (2)  had  a  knowledgeable, 
enthusiastic,  and  high-level  champion  of  the  risk  reports  who  fostered 
interdisciplinary  review  of  the  reports  and  documented  the  results  in  a  newsletter 
article.  The  results  show  that  the  facility  reduced  the  total  number  of  fall  incidents 
from  93  per  month  in  September  2002  to  53  per  month  by  February  2003.  Using 
an  estimated  average  cost  of  $1,272  per  injury14  and  assuming  conservatively  that 
one  out  of  ten  falls  results  in  injury  to  the  resident,  the  estimated  annual  savings 
for  this  facility  is  $23,000  if  this  reduction  can  be  sustained  over  time. 

Planned  evaluation — impact  on  falls  and  pressure  ulcers 

Given  the  lengthy  “start  up”  time  and  period  of  actual  use  of  the  reports,  along 
with  the  barriers  to  implementation  encountered,  it  is  too  early  to  report  results  of 
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the  evaluation  of  the  impact  of  the  risk  reports  on  rates  of  falls  and  pressure 
ulcers.  The  project  started  in  October  2001,  and  it  took  until  early  2003  to  educate 
staff  in  many  facilities,  regularly  run  the  risk  reports,  and  integrate  them  into  care 
planning.  A  minimum  of  four  to  six  quarters  is  needed  to  see  any  real  impact  on 
actual  outcome  rates. 

A  quasi-experimental  design  is  underway  to  evaluate  the  differential  impact  of 
the  risk  reports  on  risk-adjusted  fall  and  pressure  ulcer  rates  in  the  participating 
user  groups  (defined  above)  compared  with  a  matched  control  group  of  nursing 
homes  that  volunteered  for  the  project  but  were  not  selected.  It  is  assumed  that  the 
matched  group  has  similar  levels  of  motivation  to  participate  in  this  study  and 
reduce  falls  and  pressure  ulcers.  Use  of  a  control  group  selected  from  volunteer 
nursing  homes  (which  comprised  nearly  one-fourth  of  all  New  York’s  nursing 
homes)  will  allow  us  to  determine  whether  the  reports  had  an  impact  on  outcomes 
in  the  selected  nursing  homes,  despite  the  noted  differences  in  the  volunteer 
groups  from  the  nursing  home  population  in  New  York  State. 

The  control  group  was  matched  to  the  user  group  based  on  size,  a  measure  of 
rehabilitation  focus  versus  longer-term  care,  resident  population  risk  (acuity),  and 
pre-intervention  outcome  rates.  This  careful  selection  of  the  control  group  will 
ensure  that  the  comparisons  in  outcome  rates  over  time  are  not  confounded  by 
facility  characteristics,  changes  in  resident  mix,  or  other  factors  impacting 
outcomes. 

The  final  project  evaluation  will  include  followup  interviews  with  high  and 
low  use/integration  nursing  homes.  The  interviews  will  specifically  address 
staffing  levels  and  turnover  during  the  study  period  as  well  as  presence  of  a 
champion(s).  In  addition,  a  resident  safety  culture  survey  will  be  administered  to  a 
sample  of  high  and  low  use/integration  facilities  to  investigate  whether  nursing 
home  safety  culture  explains  differences  in  use  of  the  risk  reports. 


Conclusion 

This  project  has  great  potential  to  demonstrate  the  power  of  clinical 
informatics  in  improving  and  sustaining  resident  safety  in  nursing  homes  and 
across  the  continuum  of  long  term  care  facilities  in  three  major  ways: 

1.  Computer-generated  risk  infonnation  will  help  LTC  staff  focus  efforts 
appropriately  to  avoid  errors  in  risk  assessment  and  care  planning. 

2.  Knowing  more  precisely  those  risk  factors  that  are  most  likely  to  lead 
to  preventable  negative  outcomes  will  allow  LTC  staff  to  implement 
system-wide  changes  and  develop  more  effective  interventions. 

3.  Predicting  the  risk  of  adverse  outcomes  will  improve  resource  and  care 
planning,  leading  to  a  more  efficient  and  cost-effective  allocation  of 
scarce  resources. 

Despite  the  great  potential  for  computer-based  clinical  decision  support 
systems  to  improve  patient  safety,  efficiency,  and  quality  of  care  in  nursing 
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homes,  this  study  has  demonstrated  that  many  barriers  remain  to  implementing 
such  systems  in  the  LTC  environment.  Our  findings  are  consistent  with  other 
studies  on  organizational  change  and  implementation  of  clinical  information 
systems  in  health  care.  The  implementation  and  effectiveness  of  informatics 
systems  depend  not  only  on  quality  and  timeliness  of  data,  but  also  on  the 
organizational  context.  ’  Many  decision  support  system  projects  fail  despite  the 
usefulness  of  the  information  and  good  intentions  of  participants.  These  failures 

38  39 

are  due  largely  to  organizational  barriers.  ’ 

The  “lessons  learned”  in  participating  facilities  regarding  conditions  for 
success  and  barriers  to  use  of  computerized  risk  reports  will  provide  new 
guidance  to  nursing  homes  nationwide  regarding  the  effective  use  of  technology 
and  clinical  informatics  to  improve  care. 
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